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The  purpose  of  this  research  project  is  to  investigate  the 
applicability  of  Fourier  and  other  shape  descriptors  to  the 
recognition  of  tactical  targets  in  aerial  imagery.  The  work  to- 
date  has  concentrated  on  a  few  test  images  supplied  by  Rome  Air 
Development  Center.  We  have  found  that  the  most  significant 
problem  in  this  particular  type  of  data  is  the  extraction  of  the 
shape  from  the  background  (segmentation). 

Two  segmentation  methods  have  been  investigated.  The  first 
involves  local  thresholding  based  on  the  grey  values  along 
detected  edges  in  the  image.  This  method  was  discussed  in  the 
Progress  Report  dated  July  24,  1981.  Another  approach  which  has 
proven  more  useful  for  the  type  of  data  tested  involves  the  loca¬ 
tion  of  potential  objects  basei  on  grey  level/edge  level  combina¬ 
tions  which  are  unique  to  the  local  background  where  they  are 
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Once  the  segmentation  is  accomplished,  the  contours  around 
each  object  are  extracted  and  normalized  Fourier  descriptors  are 
calculated.  These  normalized  Fourier  descriptors  are  then  com¬ 
pared  to  library  entries  of  known  targets  and  all  recognized  con¬ 
tours  are  identified. 

UL.  Segmentat  ion 

Consider  the  original  image  shown  in  Ficj.  1.  The  first  step 
in  automatically  recognizing  the  airplanes  in  the  image  is  to 
extract  the  contours  from  the  background.  In  infrared  imagery 
with  hot  targets,  the  object  extraction  may  become  simpler. 

The  method  which  we  have  found  most  useful  involves  two 
stages.  First,  potential  target  regions  are  found  which  differ 
from  the  more  common  background  region.  Second,  each  potential 
taraet  region  is  closely  inspected  and  any  object  unique  in  grey 
level  and  edge  content  is  extracted  from  the  background. 

A.  Potent  ia  1  Ta  met  Loc  at  i  on 

The  assumption  made  in  this  section  is  that  combinations  of 
grey  value  and  edge  value  that  occur  infrequently  over  the  entire 
image  may  be  on  potential  targets  and  should  be  investigated 
further.  The  edge  feature  picture  is  derived  so  that  the  brioht- 
ness  of  each  point  is  proportional  to  the  edge  content  in  a  win¬ 
dow  surrounding  that  point.  The  Sobel  edge  operator! I ]  is  used 
to  produce  the  edge  picture  shown  in  Fig.  2. 
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For  purposes  of  findina  unique  reqions,  the  grey  level  image 
is  quantized  to  32  levels  and  the  edge  image  is  quantized  to  8 
levels.  Each  pixel  may  now  be  considered  a  two-dimensional  vec¬ 
tor  (qrey  value  and  edge  value)  and  a  two-dimensional  histogram 
is  created  from  the  input  imaqes.  Points  in  the  original  image 
are  next  located  whose  two-value  combination  occurs  infrequently. 
Shown  in  Fig.  3  are  all  such  locations  having  a  combination 
occuring  less  than  100  times  in  the  entire  image.  The  location 
of  potential  targets  is  then  made  by  finding  concentrated  clus¬ 
ters  of  such  points. 

This  is  done  by  averaging  Fig.  3  over  a  circular  window  of 
diameter  14  and  then  detecting  local  maxima  in  the  averaged 
result.  The  result  of  averaging  Is  shown  in  Fig.  4  and  the  local 
maxima  are  shown  in  Fig.  5.  These  points  represent  the  potential 
target  locations  in  the  original  image  (Fiq.  I).  iMote  that  all 
12  aircraft  as  well  as  additional  objects  are  marked  as  potential 
targets . 

ti.  Eniential  Target  Extract iaa 

The  next  step  in  the  process  is  to  select  the  grey  level  and 
edge  images  in  a  small  (100x100)  region  surrounding  each  poten¬ 
tial  target  point.  The  laroe  airplane  at  the  bottom  center  of 
Fio.  I  is  shown  in  Fig.  6  (a)  and  (b). 

Segmentation  of  the  object  from  the  background  is  obtained 
by  assuming  at  least  part  of  the  object  is  in  the  center  of  the 
imaoe  and  generating  a  two-dimensional  histooram  of  the  object 


Fiq.  3.  All  locations  in  Fig.  I  have  a  grey  level/edge  level  com 
bination  which  occurs  less  than  100  times  in  the  entire  image. 
These  represent  unique  points  which  may  indicate  target  loca- 
t ions . 


The  average  of  Fig.  3  using 


g.  6(a).  A  100x100 
lich  has  its  center 
cations  in  Fig.  5. 
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and  of  the  background  under  the  constraint  of  spatial  continuity. 
This  might  be  considered  a  form  of  region  qrowing.  The  image  Is 
divided  into  annuli  defined  by  concentric  circles,  with  origin  at 
the  center  of  the  imaae,  as  shown  in  Fig.  7.  The  radius  of  each 
circle  is  such  that  there  is  an  equal  number  of  points  in  each 
annulus.  Each  annulus  contains  480  points.  Sixteen  concentric 
circles  at  480  points  per  annulus  covers  most  of  the  100x100 
pixel  image.  A  two-dimensional  histogram  of  the  number  of 
occurences  of  each  combination  of  grey  level  and  edge  in  each 
annulus  is  generated  using  32  levels  of  grey  and  8  levels  of 
edge. 


In  order  to  determine  the  histograms  of  the  object  and  back¬ 
ground,  each  two-dimensional  vector  count  is  observed  as  a  func¬ 
tion  of  radius.  The  initial  estimates  of  the  object  and  back¬ 
ground  histograms  are  determined  by  assuming  that  every  vector 
which  occurs  in  the  innermost  annulus  and  does  not  occur  more 
than  five  times  in  the  outermost  annulus  is  an  object  vector.  Any 
vector  which  occurs  more  than  five  times  in  the  outermost  annulus 
is  a  background  vector.  The  object  (background)  histoqram  is 
incremented  by  the  count  of  all  the  pixels  in  the  current  annulus 
having  that  object  (background)  vector.  An  initial  estimate  of 
the  extent  of  the  object  is  calculated  as  the  boundary  of  the 
annulus  for  which  ninety  percent  of  the  object  vectors  do  not 
occur.  Next  all  feature  vectors  that  do  not  occur  inside  the 
maximum  object  radius  are  classified  as  background. 


Fig.  7.  The  16  annuli  used  in  the  segmentation  (region  growing) 
procedure  to  extract  the  potential  target  from  the  background. 
Each  annulus  contains  480  pixels. 
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At  this  point  a  major  class  of  vectors  normally  remain  unas¬ 
signed*  those  vectors  which  do  not  occur  in  either  the  center  or 
outermost  region  but  begin  occuring  before  the  maximum  object 
radius.  This  remaining  class  of  vectors  is  assigned  to  either 
the  background  or  the  ooject  histogram  using  the  "majority 
nearest  neighbor  scheme"  described  next.  Proceed  from  the  inner¬ 
most  annulus  to  the  annulus  corresponding  to  the  maximum  object 
radius.  Whenever  an  unclassified  feature  vector  is  encountered, 
calculate  the  number  of  object  pixels  and  background  pixels  that 
map  into  same  feature  space  neighborhood.  The  neighborhood  is 
defined  to  be  the  vectors  adjacent  to  the  vector  in  consideration 
in  the  two-dimensional  feature  space.  The  feature  space  consists 
of  data  collected  from  the  entire  object.  Assign  the  unclassi¬ 
fied  vector  to  be  object  or  background  based  on  the  larqer  of  the 
two  counts.  Update  the  object  or  background  histogram  to  include 
all  pixels  having  this  feature  vector. 

Once  all  vectors  are  assigned,  the  image  is  again  scanned 
and  all  points  givina  vectors  occuring  in  the  background  or  out¬ 
side  the  object  radius  are  set  to  zero.  Thus  the  object  is  seg¬ 
mented  from  the  background. 

The  results  of  this  process  operating  on  Fig.  6  is  shown  in 
Fio.  8.  Note  the  broken  wing  due  to  the  shadow  in  the  oriqinal 
imaoe.  We  have  found  that  two  stages  of  expansion  (dilation)  as 
shown  in  Fig.  9  are  required  to  join  such  broken  pieces.  Two 
staoes  of  contraction  are  then  used  to  restore  the  object  to  its 
original  size.  The  result  is  shown  in  Fig.  10. 
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The  contour  is  traced  around  the  segmented  object  and  is 
shown  in  Fig.  I  I .  When  the  same  process  as  described  in  this 
section  is  applied  to  all  the  potential  tarqet  points  shown  in 


Fig.  5,  the  contours  shown  in  Fig.  12  result.  These  are  the  con 
tours  which  are  then  input  to  the  Fourier  descriptor  recognition 
system  for  classification. 


12.  All  contours  extracted  from  the  original  using 
:ial  target  locations  shown  in  Fig.  5.  These  repr 
jrs  to  be  classified  by  the  Fourier  descriptor  met 
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T 1 1  -  Contour  Recognition 

Each  contour  to  be  classified  is  first  filtered  ( 3 %  of  total 
contour  length)  to  remove  noise.  It  is  then  resampled  to  a  power 
of  2  to  allow  a  Fast  Fourier  Transform.  The  Fourier  descriptor 
is  calculated  and  is  normalized  with  respect  to  scale,  transla¬ 
tion,  rotation,  and  starting  point  of  trace.  Ref.  [21  contains 
details  of  the  normalized  Fourier  descriptor  algorithm. 

The  normalized  Fourier  descriptor  is  then  compared  to  stored 
library  entries  of  various  targets  (in  this  case,  airplanes). 

For  the  tests  described  here,  the  library  airplane  contours  were 
extracted  from  the  photographs  by  tracing  them  on  a  digitizing 
table.  The  actual  traced  contours  are  shown  in  Fig.  13.  Also 
shown  are  the  shapes  obtained  by  inverting  the  Fourier  coeffi¬ 
cients  which  were  actually  used  in  the  libraries.  If  the  unknown 
contour  Fourier  descriptor  matches  a  library  entry  to  within  12% 
mean-square  error,  the  contour  is  classified  as  an  airplane.  The 
percent  mean-square  error  is  found  by  dividing  the  sum  of  the 
mean-square  differences  between  each  Fourier  coefficient  in  the 
unknown  and  the  library  entry  by  the  sum  of  the  squares  of  the 
unknown  coefficients  and  multiplying  by  100%.  The  contours  clas¬ 
sified  as  airplanes  are  shown  in  Fig.  14.  Eleven  of  the  twelve 
airplanes  were  classified  correctly  giving  a  classification  accu¬ 
racy  of  91%.  One  false  alarm  was  also  classified  as  an  airplane. 


Fir?.  13.  The  first  column  shows  the  four  library  airplane0 
obtained  by  using  a  digitizing  table.  The  second  column  snowi  the 
shape  obtained  by  inverting  the  first  16  Fourier  harmonics  01  the 
contours  shown  in  the  first  column. 
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Although  no  effort  has  been  made  to  optimize  execution  time 
of  the  algorithms  implemented*  some  indication  of  the  procession 
time  required  will  be  given.  The  times  given  are  for  a  512x512 
picture  using  a  VAX  11/780  computer* 

Operation  lime. 


Der i ve 

edge  feature  picture  (Sobel) 

Fio.  2 

1  18 

sec 

Locate 

potential  targets  (postarg) 

Fig.  3 

26 

sec 

(average ) 

Fig.  4 

290 

sec 

( mountain ) 

Fig.  5 

1  38 

sec 

Extract 

potential  targets  Figs.  6-12 

1202 

sec 

Classify  all  contours 

Fig.  13 

22 

sec 

Total  Time 

1  796 

sec 

Thus  of  the  30  minutes  required  to  process  the  picture,  only 
22  seconds  were  actually  used  in  the  classification  of  the  con¬ 
tours  and  all  the  remaining  time  was  used  to  extract  the  contours 
from  the  background. 
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As  a  second  example  of  this  method,  the  photograph  shown  In 
Fig.  15  was  used  as  an  input  to  the  same  recognition  system. 

Also  shown  in  Fig.  15  is  the  edge  picture.  The  unique  grey 
level/edge  level  points  are  shown  in  Fiq.  16(a)  and  the  averaae 
of  these  points  is  shown  in  Fig.  16(b).  The  local  maxima  of  the 
image  in  Fig.  16(b)  represent  potential  target  points  which  in 
this  case  are  all  accurate.  The  contours  extracted  as  potential 
taraets  are  shown  in  Fig.  17(a)  and  the  contours  actually  recog¬ 
nized  as  airplanes  are  shown  in  Fig.  17(b).  All  are  classified 
correct ly. 
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Fig.  15.  (a)  Left*  Anot 
256x512  pixels,  256  are 


rig#  lo#  (a)  Left*  All  locations  In  Fig#  15  having  a  grey 
level/edge  level  combination  which  occurs  less  than  100  times  in 
the  entire  image.  (b)  Right*  The  average  over  Fig.  16(a)  using 
circular  window. 
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We  have  demonstrated  the  feasibility  of  extracting  contours 
from  aerial  imagery  and  classifying  these  contours  using  a 
Fourier  descriptor  method.  The  most  demanding  task  for  images 
produced  using  sensors  in  the  visible  wavelengths  is  the  extrac¬ 
tion  of  the  contours  from  the  background  (segmentation).  We  have 
demonstrated  a  method  based  on  finding  unique  regions  in  qrey 
level/edge  level  space  which  has  this  capability  for  the  test 
images  shown.  The  computer  time  required  to  do  this  is  quite 
high  (30  minutes  for  one  512x512  image).  However  the  techniques 
described  have  parallel  implementations  which  would  allow  very 
rapid  speeds  using  a  special  purpose  image  processing  system. 

The  Fourier  descriptor  method  works  very  well  when  the 
extracted  contours  are  accurate.  However  the  presence  of  broken 
contours  or  partially  wrong  contours  (such  as  caused  by  shadows) 
can  give  rapid  deterioration  in  Fourier  descriptor  performance. 

The  following  directions  are  suggested  for  future  research* 

(1)  Improve  the  segmentation  and  contour  extraction  procedure  oy 
investigating  the  effects  of  parameter  variations  and  the 
results  of  additional  runs  on  various  types  of  data. 

(2)  Investigate  the  implementation  of  the  segmentation  method  on 
a  parallel  processing  computer  architecture  and  estimate  the 
processing  time  required  using  the  proposed  machine. 

(3)  Improve  the  Fourier  descriptor  recognition  method  to  include 
the  combining  of  two  partial  contours  to  allow  comparison  to 
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one  complete  library  contour.  This  would  allow  the  segmenta¬ 
tion  method  to  output  a  broken  object  and  have  accurate 
recognition  still  occur. 

(4)  Compare  the  use  of  normalized  moment  descriptors  with  Fourier 
descriptors  for  accuracy  on  this  type  of  data,  and  if  useful, 
define  a  combined  feature  set  which  includes  the  best  proper¬ 
ties  of  both  methods. 


N)  R.  0.  Duda  and  P.  E.  Hart,  EallfiXQ  Class! fl cation  and  Scene 
Ana lys is  T  John  Wiley,  New  York,  1973. 

(2)  T.  P.  Wallace  and  P.  A.  rtintz,  "An  Efficient  Three- 


Dimensional  Aircraft  Wecognition  Algorithm  using  Normalized 
Fourier  Descriptors,"  Computer  Graph! cs  and  Image  Process ! n 
Vol .  13,  pp.  99-126,  1980. 
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